Challenges in impact evaluation of development interventions: opportunities and limitations for randomized experiments

In recent years debates on as well as funding of impact evaluations of development interventions have flourished. Unfortunately, controversy regarding the promotion and application of randomized experiments (RE) has led to a sense of polarization in the development policy and evaluation community. As some proponents claim epistemological supremacy of REs (with respect to attribution) the counter reaction among others has been rejection. Needless to say, such extreme positions are counterproductive to reaching a goal that is commonly endorsed: to learn more about what works and why in development. This paper discusses the prospects and limitations of REs from the perspective of three categories of challenges in impact evaluation: delimitation and scope, attribution versus explanation, and implementation challenges. The implicit lesson is twofold. First of all, the question ‘to randomize or not to randomize’ is overrated in the current debate. Limitations in scope, applicability as well as implementation will necessarily restrict the use of REs in development impact evaluation. There is a risk that the current popularity of REs in certain research and policy circles might lead to a backlash as too high expectations of REs may quicken its demise. More importantly, given the nature and scope of the challenges discussed in the paper, more energy should be devoted to developing and testing ‘rigorous’ mixed method approaches within a framework of theory-driven evaluation.

Dealing with Stakeholder Values in the Evaluation of Development Programs. A Methodological Framework for Mid Term Evaluation

In mid term program evaluations evaluators are often confronted with the double task of retrospectively judging the program's merit and worth while at the same time advising decision makers concerning future adjustments in courses of action. In such cases, it can be argued that it is particularly important that evaluators take into account the divergent views and needs of different stakeholder groups. In principle, program theory evaluation can constitute a sound basis for dealing with the double objective of retrospective judgment and proactive program improvement. However, as argued in the paper, current approaches in program theory evaluation may not be sufficiently equipped to systematically deal with divergent stakeholder values. Taking into account lessons from the literature on stakeholder values in evaluation, an alternative methodological framework is presented. The framework combines program theory evaluation with elements of multicriteria decision aid. An example is used to illustrate the framework.

Impact Evaluations and Development: Nonie Guidance on Impact Evaluation

In international development, impact evaluation is principally concerned with final results of interventions (programs, projects, policy measures, reforms) on the welfare of communities, households, and individuals, including taxpayers and voters. Impact evaluation is one tool within the larger toolkit of monitoring and evaluation (including broad program evaluations, process evaluations, ex ante studies, etc.).The Network of Networks for Impact Evaluation (NONIE) was established in 2006 to foster more and better impact evaluations by its membership -- the evaluation networks of bilateral and multilateral organizations focusing on development issues, as well as networks of developing country evaluators. NONIE's member networks conduct a broad set of evaluations, examining issues such as project and strategy performance, institutional development, and aid effectiveness. By sharing methodological approaches and promoting learning by doing on impact evaluations, NONIE aims to promote the use of this more specific approach by its members within their larger portfolio of evaluations. This document, by Frans Leeuw and Jos Vaessen, has been developed to support this focus.For development practitioners, impact evaluations play a keyrole in the drive for better evidence on results and development effectiveness. They are particularly well suited to answer important questions about whether development interventions do or do not work, whether they make a difference, and how cost-effective they are. Consequently, they can help ensure that scarce resources are allocated where they can have the most developmental impact

Methodological Challenges in Impact Evaluation: The Case of the Global Environment Facility (GEF)

In this paper, we explore some of the methodological challenges that evaluators face in assessing the impacts of complex intervention strategies. We illustrate these challenges, using the specific example of an impact evaluation of one of the six focal areas of the Global Environment Facility; its biodiversity program. The paper discusses how theory-based evaluation can provide a basis for meeting some of the challenges presented.

Measuring Outcome and Impact of Small Development Projects: Lessons from the Evaluation of a Training Project on Low External Input Agriculture in Guatemala

Impact studies at the level of the beneficiary are not very common in the case of small-scale development projects. In this paper we will present the case of an evaluation study on a training project in Low External Input Agriculture in Guatemala. The evaluation design is based on a simple quasiexperimental design and complemented by qualitative methods of data collection. We want to illustrate that the type of mixed method evaluation used in the case can constitute a useful alternative to study the outcome and impact of small-scale development interventions given their specific constraints of money, expertise and time.

Poverty, institutions and interventions: a framework for an institutional analysis of poverty and local anti-poverty interventions

At a time when technological innovations are making our world increasingly smaller and our production systems are becoming increasingly more efficient, the benefits of economic growth and development as a whole have not been able to reach all of society. Indeed, many poor countries, characterised by their disadvantageous position in the global society and continuously plagued by weak governments, internal strife and natural disasters have missed out on many of the benefits of growth and development. Within countries that do gain advantage from the various developments of globalisation, significant groups continue to be excluded from the benefits of this new-found prosperity. It is quite significant that a generalised conclusion such as this is still a reality at the turn of the century, despite decades of national and international effort to promote development and combat poverty.

Making M&E More ‘Impact?oriented’: Illustrations from the UN

In international development, impact evaluation (IE) is becoming more and more an institutionalised practice. This article starts out by addressing the question of what institutionalisation of IE means and how it could work. Subsequently, the article explores common challenges in monitoring and evaluation functions in the UN system related to the supply of (and to a lesser extent demand for) evidence on impact. Rather than looking for solutions to these challenges in the practice of IE, the article explores the issue of how to improve non?IE monitoring and evaluation practices. On the basis of the identified challenges three categories of solutions are discussed: improving the quality of impact?related evidence at activity and project level, strengthening the causal logic underlying interventions, and strengthening the aggregation and synthesis of evidence. Finally, the article presents some illustrative examples of the latter two categories of solutions

The effect of microcredit on women's control over household spending in developing countries: a systematic review

Background: Over the past three decades, microfinance activities have spread across the globe, reaching tens of millions of poor households with tailored financial services. Microfinance can best be described as a field of intervention rather than a particular instrument. Initially, microfinance usually meant microcredit for working capital and very small investments, but increasingly it has been broadened to include savings/deposits, a limited range of micro-insurance and payment services (including micro-leasing) as well as a somewhat broader range of credit products for more substantial investments. In this study we focused on microcredit activities, constituting the bulk of microfinance activities across the globe. Microcredit activities have affected the lives of clients and others in multiple ways. The most frequently reported types of effects of credit at individual, enterprise and household level are the following: income, expenditure smoothing, and poverty alleviation effects; business growth and employment effects; schooling effects; and effects in terms of women's empowerment. Despite the diversity in microcredit schemes, many share two characteristics: they target poor women and often rely on some type of group-based lending. Women's empowerment in relation to microcredit has been studied extensively within the context of this type of microcredit scheme. Most of these studies have been carried out in the context of microcredit group schemes in South Asia. It has been argued that access to microcredit can foster changes in individual attitudes of women (e.g. increased self-reliance), power relations within the household (e.g. control over resources) and social status. An important dimension of empowerment concerns women's control over household spending. The main assumption is that by providing credit to poor women, their direct control over expenditures within the household increases, with subsequent implications for the status of women and the well-being of women and other household members. Women's control over household spending is a frequently recurring aspect analyzed within the context of microcredit interventions, which allows us to study whether microcredit targeted at women affects women's control over household spending decisions and the circumstances in which this occurs. Despite the central and recurrent role across studies of this aspect of women's empowerment in relation to microcredit activities, there has been no previous review on this topic. The growing importance of microcredit has resulted in a vast number of research and evaluation studies, including impact studies. Consequently, the microfinance literature harbors a substantial number of synthesis studies which discuss a set of microcredit interventions and aim to generate overall conclusions on their effects. However, most of these studies face limitations in terms of depth of empirical assessment and the extent to which the identified effects can be attributed to microcredit. Moreover, methodological principles regarding comprehensive searches and principles of selection, coding, extraction and aggregation are often lacking in review studies. Partial exceptions are three recent systematic reviews which all differ in scope from the present one (Stewart et al., 2010; Duvendack et al. 2011; Stewart et al., 2012). The reviews respectively focus on microfinance (credit and savings) in Sub-Sahara Africa, microcredit worldwide, and microfinance worldwide (credit, saving and leasing). Overall, these reviews suggest that the effects of microcredit on women's empowerment are at best mixed. In part this can be explained by the heterogeneity in microcredit interventions, contexts and target groups. However, the existing reviews did not use statistical meta-analysis to synthesise evidence of effects, nor context-mechanism-outcome synthesis to understand the variation in effects. Objectives: The main objective of this study was to provide a systematic review of the evidence on the effects of microcredit on women's control over household spending in developing countries. More specifically, we aimed to answer two related research questions: 1) what does the impact evaluative evidence say about the causal relationship between microcredit and specific dimensions of women's empowerment (women'ss control over household spending); and 2) what are the mechanisms which mediate this relationship. We prioritise depth of analysis over breadth, thus the scope of this review is narrower than previous systematic reviews on microfinance (Stewart et al., 2010; Duvendack et al. 2011; Stewart et al., 2012). We focused on specific aspects of women's empowerment which allowed us to combine statistical meta-analysis and realist (context-mechanism-outcome) synthesis. Criteria for considering studies for this review: We included studies that analyzed the effects of microcredit schemes targeting poor women in low and middle income countries, as defined by the World Bank. Studies that did not include analysis on microcredit and the effect on one or more dimensions (specified in main body of the report) of women's control over household expenditures were excluded. Studies which gave evidence of addressing the attribution problem either through randomised design, quasi-experimental matching, or regression analysis, were included. In practice, women's control over household spending (as a key dimension of empowerment) is influenced by many different factors. By focusing on those studies which explicitly addressed the challenge of separating the effect of microcredit from other influencing factors, we developed what we consider to be the most credible evidence base for drawing conclusions about the effects of microcredit on women's control over household expenditures in different contexts. SEARCH STRATEGY We conducted a comprehensive search covering all relevant academic databases, internet search engines and web sites with published and unpublished research, and also carried out extensive manual searches of books and additional journals not included in electronic data bases (searches were concluded on December 31, 2011). We used back-referencing from recent studies as well as citation-tracking to identify additional relevant studies. Finally, authors of studies which we were unable to retrieve were contacted. In addition, we contacted experts on microcredit and women's empowerment for additional references which we might have missed. Search strategies in databases and journals were adapted for each source. Where possible we used the existing keyword indices of particular databases. In addition, we applied our own list of combinations of keywords covering all relevant terms relating to the independent variable (i.e. credit and its variations) and the dependent variable (i.e. dimensions of women's control over household spending, empowerment). Data collection and analysis: From the different searches we identified an initial number of 310 papers that were selected for full text examination. Eventually, 29 papers were retained for further analysis, corresponding to 25 unique studies. These 25 independent findings were included in the synthesis. However, based on a systematic risk of bias assessment we found that more than half of the included studies had high threats to internal validity. Moreover, only about half of the studies show a clear and coherent link between a theoretical framework on microcredit and women's control over household spending and empirical data analysis. It should be noted that reviewing and synthesizing quantitative results from studies is only one side of the coin. The other side is to understand what makes them work, or what prevents them from working. Consequently, we conducted a qualitative synthesis of the included studies, which focused on identifying the mechanisms which underlie the causal relationship between microcredit and women's control over household spending. RESULTS The results of the meta-analysis indicated that the effect sizes from experimental studies examining effects of microcredit on women's control over household spending are not statistically significantly different from zero. The effects from quasi-experimental studies are statistically insignificant overall, and at best of small magnitude for those studies assessed of being of high risk of bias. We conclude that there is no consistent evidence for an effect of microcredit on women's control over household spending. In the qualitative analysis, using Coleman's (1986, 1990) typology of mechanisms, we identified five different situational mechanisms and eight different action-formation mechanisms. Due to the combination of substantial heterogeneity in contexts (e.g. existing gender relations) and interventions (e.g. microcredit versus microcredit and additional services), and the lack of information in the studies on this heterogeneity, it was not possible to go beyond the identification of mechanisms, in terms of generating empirically tested articulated theories of change which are representative beyond a specific study context. Authors' conclusions: In line with three recent other reviews on microfinance (Stewart et al., 2010; Duvendack et al., 2011; Stewart et al. 2012) we found that the microcredit evidence base is extensive, yet most studies are weak methodologically. From those studies deemed comparable and of minimum acceptable quality, we concluded that overall there is no evidence for an effect of microcredit on women's control over household spending. Women's control over household resources constitutes an important intermediary dimension in processes of women's empowerment. Given the overall lack of evidence for an effect of microcredit on women's control over household resources it is therefore very unlikely that, overall, microcredit has a meaningful and substantial impact on empowerment processes in a broader sense. While impacts on empowerment may appear to have occurred in particular studies, the high risk of bias of studies providing positive assessments suggests that such findings are of limited validity. Our conclusions on the effects of microcredit on empowerment are also in line with previous systematic reviews by Duvendack et al. (2011) and Stewart (et al. 2010) who report to a limited extent on empowerment effects. Consequently, there appears to be a gap between the often optimistic societal belief in the capacity of microcredit to ameliorate the position of women in decision-making processes within the household on the one hand, and the empirical evidence base on the other hand. However, our review markedly differs from previous reviews in two regards. First, we specifically focused on microcredit and women's empowerment captured through women's control over household expenditures. Second, as a result of this narrower focus, we were able to conduct statistical meta-analysis and extract behavioral mechanisms which can help to explain why and how microcredit can make a difference. The advantage of our approach was that the identified mechanisms all stem from studies which show evidence of addressing the attribution problem. Consequently, we can be quite confident of the insights that they provided on the effects of microcredit on women's control over household spending for particular populations of microcredit female clients and their families. Those studies that showed evidence of addressing the attribution problem were relatively weak on underlying theory. Moreover, they often lacked essential information such as the nature of the intervention and how it related to empowerment (e.g. how solidarity groups affect empowerment processes) or the slowly evolving gender relations in different contexts (e.g. the evolution of societal norms and the relationship with power relations in the household). A next logical step would be to undertake a systematic review of qualitative studies which often provide rich and context-specific information on microcredit and women's decision-making power in the household. Such a review should ideally build on the mechanisms identified in the present review and would bring us closer to uncovering credible theories of microcredit and the circumstances in which it may change women's decision-making power

An IGF-I promoter polymorphism modifies the relationships between birth weight and risk factors for cardiovascular disease and diabetes at age 36

OBJECTIVE: To investigate whether IGF-I promoter polymorphism was associated with birth weight and risk factors for cardiovascular disease (CVD) and type 2 diabetes (T2DM), and whether the birth weight – risk factor relationship was the same for each genotype. DESIGN AND PARTICIPANTS: 264 subjects (mean age 36 years) had data available on birth weight, IGF-I promoter polymorphism genotype, CVD and T2DM risk factors. Student's t-test and regression analyses were applied to analyse differences in birth weight and differences in the birth weight – risk factors relationship between the genotypes. RESULTS: Male variant carriers (VCs) of the IGF-I promoter polymorphism had a 0.2 kg lower birth weight than men with the wild type allele (p = 0.009). Of the risk factors for CVD and T2DM, solely LDL concentration was associated with the genotype for the polymorphism. Most birth weight – risk factor relationships were stronger in the VC subjects; among others the birth weight – systolic blood pressure relationship: 1 kg lower birth weight was related to an 8.0 mmHg higher systolic blood pressure CONCLUSION: The polymorphism in the promoter region of the IGF-I gene is related to birth weight in men only, and to LDL concentration only. Furthermore, the genotype for this polymorphism modified the relationships between birth weight and the risk factors, especially for systolic and diastolic blood pressure